Method and apparatus for producing and accessing composite data using a device having a distributed communication controller interface

ABSTRACT

An apparatus and method for producing composite data by a server computer with a distributed communication controller interface involves deforming a template creating a mapping relationship between co-registered data and subject data, filtering the co-registered data, and mapping this filtered co-registered data according to the mapping data. A client computer with a distributed communication controller interface requests the composite data from the server computer and transmits the subject data to the server computer. The client presents the received composite data to an operator and monitors the operators use of the composite data.

RELATED APPLICATIONS

[0001] The application is a continuation-in-part of U.S. patentapplication Ser. No. 09/382,594, filed Aug. 25, 1999, which is hereinincorporated by reference in its entirety, which is a continuation ofU.S. patent application Ser. No. 08/832,688 filed on Apr. 11, 1997 andissued as U.S. Pat. No. 5,970,499 on Oct. 19, 1999. Thiscontinuation-in-part application also claims priority to U.S.Provisional Patent Application No. 60/135,057 filed on May 20, 1999,which is also herein incorporated by reference in its entirety.

BACKGROUND OF THE INVENTION

[0002] The present invention relates to information systems and methods,and more particularly to data fusion systems.

[0003] Many applications benefit from the contemporaneous assimilationof large amounts of data. In medical, military, and commercialapplications, operators engage in procedures and make decisions based ondata describing various subjects represented by, for example, images,recorded sound, and text. Current technology has limitations presentingpersonnel with a unified view of this subject data to allow them to useall available data to make informed decisions.

[0004] For example, a physician providing medical treatment reviewsimage data acquired in multiple modalities, such as magnetic resonance(“MR”), computed tomographic (“CT”), and X-ray images, medical journalsdescribing procedures, video images, such as ultrasound, and atlasesdescribing anatomical structures. A physician therefore consults severalsources to review the data necessary to provide patient treatment. Thesesources may include multiple computer display terminals located indifferent parts of a hospital, hard copies of medical images printed onfilm archived among thousands of images in a hospital film library orremote storage site, and volumes of journals located in the stacks of ahospital library. Also, the sources of data consulted by treatingphysicians may include medical atlases containing thousands of MR and CTscans of a cadaver corresponding to photographic images ofcross-sectional slices taken of various anatomical structures.

[0005] Usually data from these atlases and other sources are notcorrelated with each other. A cadaver image in an atlas does not usuallyhave the same geometry as a patient receiving treatment, so a physicianmust mentally fuse the available data which requires correlating thedata retrieved from the various sources to develop a treatment plan orto provide information during medical procedures. The difficulties offusing available data increase if the physician must assimilate thevarious data types while rendering treatment.

[0006] The World Wide Web (“WWW”) has recently made vast amounts of datastored on local and remote computers easily accessible through agraphical computer interface. The WWW is a network of computers,connected by the Internet, sharing a common file structure and mark-uplanguage for creating files. The two most prevalent languages used tocreate multimedia WWW files are the hypertext mark-up language (“HTML”)and the virtual reality mark-up language (“VRML”). HTML is best suitedfor creating files with text and two-dimensional image data, whereasVRML is designed for creating files containing images ofthree-dimensional objects. Both languages provide an easy way to combineimage, text, and sound data in files accessible by “point-and-click,”computer mouse driven user interfaces called “browsers.”

[0007] A “browser” is a computer program that provides users access tofiles stored on the WWW. The browser displays files on a computer screenand can run programs, known as “applets,” indicating links to data inother files on the WWW by, for example, underlining text or highlightingareas of an image. By selecting the underlined text or a highlightedimage, the browser retrieves the linked data, allowing a user to viewdata stored on computers in the WWW without needing to know where theinformation is physically stored. Files can be joined using these“hyperlinks,” which give the name of the file along with an address fora computer storing the file. For example, the text or an image in a filestored on a computer in Switzerland can contain an embedded link to datastored on a computer in the United States. The WWW browser automaticallyrecognizes the linked file data type, so the linked file can be animage, an audio clip, a video, or even an executable computer program.For example, if the linked data is an audio clip, the browser will loada program that takes the audio clip and plays it through the speakers ofthe user's computer. A browser usually runs on a computer referred to asa “client,” while a computer known as a “server” hosts and produces WWWfiles requested by a client.

[0008] In particular, the WWW serves as a useful tool for navigatingthrough two- and three-dimensional image data. For example, an image canbe displayed by the browser, and different parts of the image can belinked to different files. But, for the most part, this WWW capabilityis primarily used for providing simple menus of uncorrelated dataavailable on WWW computers. For example, a WWW computer will show animage of people, cars, and boats. By clicking on the image of people, auser can go to on-line chat sessions with people, or by clicking on aboat image, a user gets information about boats.

[0009] The current technology is limited because there does not exist aninformation system that exploits the data navigation capabilities of theWWW to correlate data retrieved from diverse sources and then assimilatethe data into a useful form. For example, the tools available forinformation gathering in the WWW environment include database searchengines and expert systems that assist a user in describing theinformation sought. However, these tools only retrieve filescorresponding to a particular term or pertaining to certain designatedsubject matter. The retrieved files are not correlated with one another.

[0010] There is, therefore, a need for an information system thatharnesses the power of the technology associated with the WWW and othersimilar image-based information retrieval systems to produce assimilatedcomposite data in a form that operators can readily use.

SUMMARY OF THE INVENTION

[0011] The present invention is directed to a method and apparatus forproducing and accessing composite data containing co-registered andsubject data. Co-registered data is generated, for example, byregistering data to a common coordinate system. The method forautomatically producing composite data includes several steps, performedby a server computer. The steps include: creating a mapping relationshipbetween the co-registered data and the subject data by mapping ordeforming a template to fit the subject data; filtering theco-registered data; and producing composite data by mapping the filteredco-registered data to the subject data according to the mappingrelationship.

[0012] A method consistent with this invention is also directed tosteps, performed in a client computer, including: requesting compositedata from a server computer; transmitting the subject data to the servercomputer; receiving the requested composite data from the servercomputer; presenting the received composite data to an operator; andmonitoring the operator's use of composite data.

[0013] An apparatus consistent with this invention for automaticallyproducing composite data containing co-registered data and subject dataincludes: structure for creating a mapping relationship between theco-registered data and the subject data by mapping or deforming atemplate to fit the subject data; structure for filtering theco-registered data; and structure for producing composite data bymapping the filtered co-registered data to the subject data according tothe mapping relationship.

[0014] Another apparatus consistent with the present inventionautomatically presents an operator with composite data containingco-registered data and subject data. Such an apparatus includes:structure for requesting composite data from a server computer;structure for transmitting the subject data to the server computer;structure for receiving the requested composite data from the servercomputer; structure for presenting the received composite data to anoperator; and structure for monitoring the operator's use of thereceived composite data.

[0015] Both the foregoing general description and the following detaileddescription are exemplary and explanatory and are intended to providefurther explanation of the invention as claimed.

DESCRIPTION OF THE FIGURES

[0016] The accompanying drawings provide a further understanding of theinvention. They illustrate embodiments of the invention and, togetherwith the description, explain the principles of the invention.

[0017]FIG. 1 is a block diagram of an apparatus for producing compositedata containing co-registered data and subject data consistent with thepresent invention;

[0018]FIG. 2 is a block diagram of another apparatus for producingcomposite medical data containing co-registered medical data and patientdata consistent with the present invention;

[0019]FIG. 3 is flow diagram of a method for producing composite datacontaining co-registered data and subject data consistent with thepresent invention;

[0020]FIG. 4 is a schematic diagram of user interaction with anembodiment of the present invention consistent with the block diagram ofFIG. 2;

[0021]FIG. 5 is an illustration of co-registered medical data used in anembodiment of the present invention consistent with the block diagram ofFIG. 2;

[0022]FIG. 6 is a display produced in accordance with an embodiment ofthe present invention consistent with the block diagram of FIG. 2; and

[0023]FIG. 7 is a block diagram of a facility for providing compositedata in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[0024] Reference will now be made in detail to the embodimentsconsistent with the present invention, examples of which are illustratedin the accompanying drawings.

[0025] To illustrate the principles of this invention, FIG. 1 showssubject data 100, template layer 102, deformation engine 104, mappingengine 106, map 107, filtered, co-registered data 108, search and filterengine 110, links 112, and co-registered databases 114. The term “layer”denotes a grouping of data types represented in, for example, a databaseon a server computer. To produce composite data 109, deformation engine104 deforms a template layer 102 to fit subject data 100 generating map107. When subject data 100 is multi-modal, multiple template layers canbe utilized to correlate co-registered data 114 to subject data 100 aswell as to correlate multi-modal subject data 100 to itself. A templatelayer contains reference data locations, or landmarks, used to correlatetemplate data with elements of subject data. Examples of landmarksinclude data representing points, lines, surfaces, volumes, or otherdefining features in image data.

[0026] Generally, deformation is the process of mapping one image toanother image where both images represent related structure butgenerally have different geometric proportions and orientations. Duringdeformation, mathematical transforms are applied to the images that mayperform the equivalent of bending, stretching, and/or rotating templatedata to match subject data. For example, after deformation, templatedata in the form of a volume image of a generalized model of the humanbrain is manipulated to relate the size, shape, and orientation of theanatomical structure in this model to the subject data, the actualanatomy of a patient receiving treatment. There are many techniquesavailable for deforming one set of data to fit a target data set,including rule-based morphology, correlation of selected landmarks ineach data set, and a technique fusing selected landmarks and image data.One example of such technique appears in U.S. Pat. No. 6,009,212 toMiller et al., which is herein incorporated by reference. Similartechniques are also suitable for generating co-registered dataconsistent with the present invention.

[0027] Once template layer 102 is deformed to fit subject data 100, amapping relationship, map 107, is established whereby mapping engine 106maps co-registered data 114 to subject data 100 producing composite data109. Co-registered data 114 represents a knowledge base providingsupplemental information about structures contained in subject data 100.The co-registered databases and template layer share a common coordinatesystem, so a data element representing a position in one co-registereddatabase is correlated with a data element representing that sameposition in each of the other co-registered databases. In an embodimentconsistent with the present invention, co-registered data 114 isco-registered using, for example, the deformation techniques describedabove. The mapping relationship obtained from deforming template layer102 to fit subject data 100 correlates the co-registered databasecoordinate system with a subject data coordinate system. Implementingthis mapping relationship, mapping engine 106 relates points in thesubject data coordinate system to corresponding points in theco-registered database coordinate system, providing a dynamic connectionbetween subject data 100 and co-registered databases 114.

[0028] Search and filter engine 110 controls which elements ofco-registered data 114 are mapped to subject data 100 and presented toan operator. Search and filter engine 110 can allow mapping engine 106to map all or a subset of co-registered data 114 to subject data 100.Links 112 specify relationships among data elements across co-registereddatabases 114 which are used by search and filter engine 110 toassimilate co-registered data according to a service request by anoperator including, for example, an indication of a region or regions ofinterest in subject data 100. The links 112 may be formed using anappropriate database indexing strategy to assign key-word or conceptsearch tags to associated data elements.

[0029]FIG. 2 shows an embodiment consistent with the present inventionfor producing composite patient data for medical treatment. Collectedpatient data 200, includes, for example, MR image 236, CT image 238, andlandmarks 240. Template layer 202 includes corresponding MR image 230,CT image 232, and landmark 234 templates. This embodiment also includesa deformation engine 204, mapping engine 206, search and filter engine210, links 212, and co-registered data 214.

[0030] Co-registered data 214 includes a medical atlas 228, text 226,computer programs 225 (such as applets), labels and landmarks 224,images 222, audio and video clips 220, links to computers located on theWWW 218, and treatment plans 216. Co-registered data can also includeco-registered subject data. The forgoing list of co-registered datatypes is only provided as an example of the types of data that areuseful in practicing this invention in the context of providing medicaltreatment. Persons of ordinary skill will recognize that many other datatypes may also be useful. One of ordinary skill in the art will alsorecognize that two or more of the co-registered data types may reside ina single database.

[0031]FIG. 3 is a flow diagram of a method for producing compositemedical data consistent with the invention. An operator of a systemconsistent with this invention wishing to have composite data formedical treatment first identifies and possibly collects patient data200 using a client computer (step 300). The collected patient datashould have a modality (e.g., CT, MR, or X-ray) and protocol (e.g.,slice thickness and resolution) that is compatible with a template intemplate layer 202.

[0032] The client computer then generates request 246 for composite data244 (step 304). Request 246 includes, for example, an operatoridentifier, security screening information, and treatment information.The client computer also transmits or enables transmission of (from aradiological database, for example) collected patient data 200 andassociated filtering context 242 to the server computer (step 306).

[0033] Responding to client computer request 246 for composite data 244,the server computer selects a template conforming with this request(step 308). Example template, MR 230, employed by the present inventionwhen used for medical treatment includes a three-dimensional MR scan ofthe same slice thickness, resolution, and collection protocol as thepatient MR dataset 236. Associated with the selected template imagery isa set of landmarks 234 identifying anatomical structures in a region ofinterest for a particular surgical procedure. Next, deformation engine204 fits the selected template to patient data received from the clientcomputer. The process of deforming the selected template to fit thepatient data 200 creates a mapping relationship (map 248) relatingtemplate data space to a patient data space coordinate system (step310). Mapping engine 206 also uses map 248 to relate a point in thepatient data space coordinate system to an element of co-registered data214. Once the mapping relationship is determined by deforming theselected template, all co-registered data 214 can be mapped to patientdata 200. Note that if multi-modal patient data is used with multiplecorresponding templates, multiple maps 248 can be constructed that canthen be used to correlate the multi-modal patient data with each otherin addition to correlating co-registered data 214.

[0034] Search and filter engine 210 controls how much of co-registereddata 214 is included in the composite data. One reason for such controlis that certain data types and/or data elements in co-registered data214 may not be relevant for some medical treatment. Search and filterengine 210 responds to filtering context 242 and selects co-registereddata elements as appropriate for this filtering context (step 312) usinglinks 212 to identify related data elements. The filtering context canbe derived from data provided by the client computer during the initialrequest for composite data (step 304). A filtering context can also bederived from previously stored profiles or histories. The server thenproduces composite data using mapping engine 206 and map 248 to combinepatient data and filtered, co-registered data (step 314), producingcomposite data 244. The server then transmits composite data 244 and map248 to the client computer for presentation to an operator (step 318).The operator navigates the composite data by specifying a region ofinterest in the patient data using a browser interface (step 320). Theoperator may also use the browser interface to select highlighted textor other specified segments of the composite data, activating a link toa particular region of interest in the patient data.

[0035] Map 248 also allows an operator to access additional compositedata (step 322). The server receives a request for additionalco-registered data and preferably an associated position in the patientdata coordinates from the client and subsequently retrieves andtransmits additional co-registered data using mapping engine 206, map248, search and filter engine 210, and links 212 to additionalco-registered data 214. (repetition of steps 312-320).

[0036] An embodiment consistent with the present invention for use inthe medical field links a radiologist's report to radiological imageryof a patient. Preferably, in an area of the radiologist's text reportstating, for example, “in the left parietal-occipital region is a 1.2 cmhypodense lesion with an irregular border that does not enhance oncontrast but is hyperintense on T2 . . . ”, selecting the highlightedword “lesion” activates a link to the patient imagery which highlightsthe particular sub-region discussed in the report text. Likewise, if theoperator selects the lesion site in the displayed patient imagery, thelink will be activated to display the section or sections of the textreport that discuss the selected region of interest.

[0037] Although the foregoing description of embodiments of the presentinvention specifically allocate certain operations to client and servercomputers, one of ordinary skill in the art will recognize that thedistribution of specific tasks between client and server computers canvary based on application requirements. Moreover, embodiments of thepresent invention with several client or server computers are within thescope of this invention. Furthermore, it is also consistent with thepresent invention that the client and server tasks can be performed in asingle computer.

[0038] In an embodiment consistent with the present invention, agraphical user interface designed for browsing data presents compositedata 244 to the operator. The interface can be executed on networkedcomputers. Computer program code that can be adapted to perform thisbrowsing function includes Internet browsers designed to navigate theWWW, such as Netscape's Navigator and Microsoft's Explorer, andequivalent programs that support links to data stored on networkedcomputers.

[0039] Communication among the devices and with the Internet iscontrolled by the surgeon or other staff within the operating room usingthe web-like interface or browser. Thus, operating room staff havecontrol over information allowed into and out of the operating room bythe switch, to insure patient privacy and security. To provide thisfunctionality, an embodiment consistent with the present inventionconnects devices within the operating room and can also communicatebidirectionally with the world beyond the operating room. Since devicesin the operating room commonly generate large data streams, oneimplementation of the present invention utilizes a broadband networkwithin the operating room. Since an implementation of the networkinfrastructure of the present invention can facilitate communicationamong information servers in addition to controlling devices such asrobots which demand accurate timing, each operating room's networkshould be able to be isolated from stray network traffic that couldinterfere with communications within the operating room. An architecturesuitable for providing such a networking infrastructure is described inU.S. Provisional Patent Application, Ser. No. 60/135,057 filed on May20, 1999, by Richard D. Bucholz, entitled “Networking Infrastructure foran Operating Room,” which is incorporated by reference herein in itsentirety.

[0040] Finally, networks as presently conceived tend to be staticconstructs, such as desktop computers connected in an office. This isoften at odds with the work flow of an operating room. Rather thanconnecting a number of devices which stay connected for long periods,the operating room is continually in flux. Networked devices may bepresent for only a portion of a particular procedure, and thepreferences of the surgeon and the demands of the procedure dictatewhich devices are employed. Therefore, an embodiment consistent with thepresent invention contemplates simplified connections wherein thenetwork or the device initiate communications automatically and promptlyupon connection. In addition, operating system(s) consistent with thepresent invention can tolerate disconnection without serious incident.Since these systems are, in many instances, life support devices,embodiments consistent with the present invention contemplate thecomponents of the system operating and being controlled despiteconnection or disconnection of a particular device from the network. Inaddition, components of the system operate and are controlled whethernetworked or not. In short, the connections are robust and faulttolerant.

[0041] An embodiment consistent with of the present invention employsthe Jini networking protocol (as developed by Sun Microsystems). TheJini network protocol allows a Jini compatible device to make and breaknetwork connections upon connection of the device to the network.Further, communications established in a Jini compatible network allowprompt sharing of information between and control of devices afterconnection. This control of networked devices is orchestrated throughstandard Internet and web technology such as hypertext transfer protocol(e.g., http over TCP/IP).

[0042] The suitability of the Jini networking protocol is made moreapparent when the inherent organization of the operating room is takeninto account. Hospitals often have operating suites with a number ofseparate operating rooms, each with substantial autonomy. Therefore,establishing a network within each operating room allows control withineach room, and multiple operating rooms may be linked through anintelligent switch in each operating room so that selectivebidirectional communications can occur between the linked operatingrooms. In addition, an embodiment of the present invention facilitatesselective bidirectional communication with the Internet.

[0043] By controlling each operating room's switch from within theoperating room, all devices in the operating room can communicate witheach other using a broad bandwidth network, and extraneous Internetnetwork traffic can be selectively prevented from entering the room.Thus, the surgeon can exercise control of devices within the operatingroom and secure patient information, while gaining access to theInternet. Of course, embodiments consistent the present invention alsocontemplates compatibility with and the utilization of other networkprotocols.

[0044] According to an embodiment consistent with the present invention,networked devices in the operating room can have a distributedcontroller that is Jini compliant and capable of communication usingstandard Jini communication protocols over a local-area or wide-areanetwork. For example, server 243 and client 245 in FIG. 2 could beequipped with a distributed communication controller interface using theJini communication protocols. Devices are controlled locally using theirown distributed controller that drives a display device that is alsoattached to the network. A client is modified by the addition of acontroller that interfaces with the network and a touch sensitive flatpanel display. The distributed controller has software for a“minibrowser” (scaled-down browser) which may be saved in read-onlymemory (ROM) along with control forms written in html. A control form isdisplayed on the display device upon startup of the client, and consistsof virtual buttons that are actuated by touch. When the user touches abutton to request a desired task, the browser activates the controllerthrough an interface so that the controller controls the client toperform the desired task. Therefore, local control of the client througha user-friendly interface is achieved using a browser in the absence ofany communication between the client and the network. Since thisembodiment of the present invention uses a browser (a web-likeinterface), the control language of the device can be changed easily,and can have a variable complexity determined by the user. For example,a display for a nurse may differ from the surgeon's display allowingeach different control capabilities.

[0045] The presence of the web-like interface enables remote control ofthe device over the network. Since the device is Jini compliant, it has,among other things, a Jini compliant controller. Therefore,communications within the operating room are established automaticallyupon plugging the device into the network. The web-like interface allowsthe device to be controlled by other devices in the operating room. Whentwo devices are connected by the network, the display of each device isprogrammed to display the control form of all connected devices. Forexample, if an MR machine is plugged into the network along with theclient, the browser of the MR machine will display the fact that othercontrol forms are available to it over the network by displaying buttonsfor each controllable device. By pressing the button marked client, theMR machine's browser will display the control form for the client, andall functions of the client can be manipulated through the control formas displayed on the MR machine's browser. This bidirectionalcommunication is established by plugging the device into a network jacklocated in the operating room, as orchestrated by the Jini networkprotocol and the device's embedded Jini-compliant controller. Accordingto this embodiment, the functions embedded in the control form are htmlcompliant and can therefore be of any form.

[0046] The present invention employs a wired local-area or wide-areanetwork, or may alternatively employ a wireless, infrared, or othersuitable network as long as the network has a bandwidth capable oftransmitting the appropriate data streams. For a simple operation,infrared communication may be adequate. Alternatively, control of asurgical robot requires a network that is robust and resistant to noise,making presently available wireless networks inappropriate. Further,presently available wireless networks may allow crosstalk betweenoperating rooms, creating potentially severe control problems.

[0047] According to another embodiment consistent with present theinvention, the devices have a controller that is Jini compliant, but thedevices are connected to each other rather than to a local-area orwide-area network. In this embodiment, the Jini protocol allowscommunications to be established between two devices without using anetwork. However, many procedures require more than two devices, andtherefore a device allowing multiple connections is needed. If nocommunication with a network outside of the operating room is desired,then a repeater can be used to create the multiple connections when morethan two devices are connected.

[0048] The present invention further contemplates use of the networkinfrastructure with a StealthStation as disclosed in U.S. Pat. Nos.5,383,454, 5,871,445, 5,891,034 and 5,851,183, and InternationalPublication Nos. WO 94/24933 and WO 96/11624, which are incorporatedherein by reference. In the StealthStation embodiment, instead of onecart holding all equipment for the StealthStation, the StealthStationconsists of two stand-alone modules.

[0049] A first module is a display unit having a high resolution touchpanel on a pole extending from an electronics cluster located oncasters. The network switch is located in the electronics cluster, alongwith the computer for a navigational system. A network jack panel isalso located in the electronics cluster, and provides a connection tothe Internet. An Internet connection is provided in each operating room,such as in the form of a telephone jack with a modem.

[0050] In the second module of the StealthStation embodiment of thepresent invention, at least one camera is attached to a long armconnected to an electronics cluster located on casters. The cameracommunicates with the display system through the network. Therefore, anetwork cable extends from the camera electronics cluster to the networkswitch in the display module, and another cable extends from the displaymodule to the Internet wall jack. Any other devices used with thenetwork are connected to the network switch located in the base of thedisplay unit. Thus, the StealthStation display unit is the hub of theoperating room network. Alternatively, for example, the network switchcan be wall-mounted in the operating room so that the StealthStationneed not contain the network switch.

[0051] Notably, new technology can be incorporated easily into thesystem by making the new technology Jini compliant. For example, a robotcan be controlled by the networked system if its control mechanisms wereprogrammed to accept an interface consistent with the present invention,such as, for example, the Jini interface standard. New display orcontrol devices, ultrasound devices, or fluoroscopes can connect to thenetwork and transmit their images, and be controlled, by other deviceswithin the operating room. In this way, the network infrastructure ofthe present invention makes the StealthStation compatible withtechnological innovations, and fosters development of new technologieswithout need for reprogramming for each device.

[0052]FIG. 4 illustrates operator interaction associated with producingcomposite data in accordance with an embodiment of the presentinvention. FIG. 4 shows a database containing co-registered data 400 ofseveral data types, including video 402, text 404, waveforms 406,programs 407, still images 408, and segmentation labels 409; a mappingengine 410; a map 412; a set of links 414 among associated data elementsacross and within the co-registered databases; a patient data spacecoordinate system 416; and a universal (“atlas”) coordinate system 418common to all data stored in co-registered database 400. A physicianusing this embodiment of the present invention selects any point inpatient data space coordinate system 416 to retrieve co-registered data400 corresponding to the selected point.

[0053] In an example illustrating co-registered data in an embodiment ofthe present invention, still image database 408 contains MR images of ahuman head, database 406 contains recordings of waveforms produced bycertain electrical signals in the brain, video database 402 containsrecorded motion picture images of neurosurgical procedures or tutorialsfor these procedures, text database 404 contains short descriptiveparagraphs or full journal articles describing regions of the brain andrelated surgical plans, database 407 contains programs for processingimage data, and database 409 contains segmentation maps outlining brainstructures.

[0054] The patient data space coordinate system 416 is a frame ofreference for patient specific data. This coordinate system is providedby, for example, an MR of a patient's head or the surgical fieldsurrounding the patient during operation. Deformation engine 204computes a mapping relationship relating template layer data points inatlas coordinate system 418 to patient data points in patient data spacecoordinate system 416. Mapping engine 410 uses this computed mappingrelationship to transform co-registered data 400 mapped to atlascoordinate system 418 to patient data space coordinate system 416.

[0055] After mapping, a physician has available during a surgicalprocedure composite data adapted to the patient's anatomy. Thiscomposite data is a representation of (1) a patient's anatomy comprisingpatient specific data acquired before or during a medical procedure, and(2) data from one or more of the co-registered databases 400.

[0056] Map 412 provides a virtual grid overlaying the patient data spacecoordinate system 416 allowing an operator to position a pointing devicein the patient data to retrieve co-registered data. Selecting a positionin map 412 retrieves co-registered data correlated with the selectedposition by mapping engine 410 through links 414.

[0057] In one embodiment of the invention, map 412 contains a number ofpositions corresponding to the number of positions in patient data spacecoordinate system 416 detectable by a surgical navigation system (see,e.g., U.S. Pat. No. 5,383,454). A map position is selected according tothe location of a surgical probe in patient data space coordinate system416 during a medical procedure. For example, during neurosurgery asurgeon placing the probe at a patient's ventricle activates a mapposition corresponding to the probe position in the ventricle. Theactivated map position is communicated to mapping engine 410 whichqueries co-registered databases 400 for data corresponding to the mapposition in the ventricle. This corresponding co-registered data isdeformed to correlate to the patient's anatomy and combined with patientspecific data giving the surgeon composite data related to the patient'sventricle, containing more information than the patient data alone.

[0058] A further illustration of the types of data a physician mayrequire during neurosurgery or during surgical planning is shown in FIG.5. This figure contains illustrations of five data types available to asurgeon in an embodiment consistent with the present invention includinga cross-sectional image 502, a medical journal article 504,electroencephalograph waveforms 506, computer programs 507, video imagesof the brain 508, and a segmentation map identifying the differentregions of the brain 510. Because these data sources have beenco-registered to a common atlas coordinate system, a point, such aspoint 500 in the cerebellum in brain image 502 from database 408, has acorresponding point in each of the other data types in co-registereddatabase 400. For example, point 500 in text database 404 corresponds toarticle 504 on new surgical techniques involving the cerebellum. Point500 in waveform database 406 corresponds to recorded waveforms 506produced by the brain at this location. Point 500 in program database407 corresponds to applet program 507, which is used to provide enhancedvisualization of brain image 502. Point 500 in video database 402corresponds to video clips 508 of the brain at this location. Point 500in segmentation map database 409 corresponds to a point withinsegmentation map 510.

[0059] Each of these examples of data need not be acquired from thepatient currently receiving treatment. For example, the data may comefrom digital anatomical atlases, libraries of cadaver images, orresearch databases produced by projects such as the “Visible Human”research sponsored by the National Library of Medicine. Data availablein a co-registered database would include, for example: 1. AnatomicMagnetic Resonance Imaging Computed Tomography Magnetic ResonanceAngiography Ultra-Sound Slice photographic images Sulci/Gyri traces 2.Functional Positron Emission Tomography Single Photon Emission ComputedTomography Functional Magnetic Resonance images ElectroencephalographMagnetoencephalography 3. Symbolic Structure name Structure sizeStructure function Structure related text cues Structure related videocues Structure related audio cues Structure related labels HistologyMorphological data 4. Multimedia Video Footage of procedures TrainingVideos Conference, Journal Articles Statistics 5. Computer ProgramsApplets Data Analysis Automated Structural Segmentation ImageEnhancement and Visualization

[0060] These data need not be located in a single database. One ofordinary skill in the art will recognize that individual co-registereddatabases may be distributed among several databases accessible through,for example, local area computer networks or wide area computer networksconnecting local and distributed computers. The combination of thedifferent data components of the co-registered databases produces ageneric data model of human anatomy where each data element iscorrelated through a common coordinate system with corresponding dataelements of other types. When planning or providing medical treatment,the invention produces composite data linking co-registered data of ageneric model to the specific anatomy of a patient receiving treatment.Examples of other embodiments consistent with the present invention forproducing composite data for medical diagnosis, planning, and treatmentinclude, but are not limited to, the following.

[0061] 1. Diagnostic Radiology—co-registered patient magnetic resonance,X-ray, and/or computed tomography imagery are linked to text data suchas radiologists' reports, patient case history files, and relevantconference/journal articles. Sequential scans of a patient areco-registered for tracking the growth or reduction of lesions. Dataanalysis programs are linked to the composite data for computation ofquantitative data measurements for planning and monitoring of treatmentprogress. Co-registered multi-modal patient image data and relevantco-registered data are presented in a common, easy-to-use presentationscheme.

[0062] 2. Radiation Treatment Planning—Three-dimensional segmentedatlases are mapped to patient data to produce an object-based model oflesions, targets, and major organs and other critical structures. Thepatient data with associated object information is utilized by atreatment planning program for computing optimized radiation deliverystrategies from target and critical structure information.

[0063] 3. Neurosurgical Targeting—cranial patient imagery is mapped toneurosurgical atlas information containing coordinates and shapes ofsurgical targets and surrounding neuroanatomic structures. Structuralinformation is linked to audio files for use in-surgery withmicrorecording probes. Links to statistical databases provideinformation relating regions of interest to procedures and successrates.

[0064]FIG. 6 shows how one embodiment of the present invention presentscomposite data to an operator on a computer display having severalwindows. Window 604 contains patient data in the form of athree-dimensional MR brain scan. Windows 602, 606, and 608 containaxial, sagittal, and coronal photographic section data, respectively,from the Visible Human data set, which are co-registered to the patientdata by deformation engine 204. Window 610 presents co-registered atlasdata to an operator. By positioning cross-hairs 612 in Window 604 at adesired point 600, the corresponding point 600 in each of the images inwindows 602, 606, and 608 is identified automatically by the location ofcross-hairs in those windows. Here, for example, the operator selectedpoint 600 in window 604 corresponding to a region of the brain known asthe putamen. Atlas data containing views of the putamen are displayed inwindow 610 with highlighted pointers indicating the putamen in each ofthe atlas images. The operator can also choose to play a movie showingvideo images of the putamen by pushing “play movie” button 614.Alternatively the operator may select the word “putamen” in window 610and cross-hairs 612 will indicate the position of the putamen in patientdata window 604 and the Visible Human “atlas” data windows 602, 606,608.

[0065]FIG. 7 is a block diagram of a facility consistent with thepresent invention for providing composite data across a computer networkto customers under a service contract. In FIG. 7, solid lines indicate apath for both control and data flow and dotted lines indicate data flowonly. Facility 700 is preferably connected to a wide area network, suchas Internet 701, through firewall 702. Firewall 702 is a computer thatmonitors all data traffic into and out of facility 700 to preventunauthorized access to the facility. World Wide Web page 704 provides agraphical user interface to access facility 700. Facility 700 alsoincludes customer account manager 710, which controls functionsavailable to customers with service contracts authorizing access tofacility 700.

[0066] User login authentication is performed by customer accountmanager 710 and control is passed to one of three processes, servicerequest manager 706, customer database manager 708, or results manager712 depending on the service that the customer chooses. Customers thatwish to initiate a new request for composite data are passed to servicerequest manager 706. After successful completion of a composite datarequest, the customer's account is billed and the status of any pendingrequests is provided. Customers that wish to view the composite datagenerated in response to the request are passed to results manager 712.Information pertaining to a customer's account (e.g., billinginformation, changing passwords, user preferences, etc.) may be obtainedby submitting queries to customer database manager 708.

[0067] Service request manager 706 initiates service requests andcontrols the computational operations required for anatomic mapping. Theprogress of a service request is tracked and reported to customeraccount manager 710. Customer database manager 708 administers adatabase that contains customer account data (not shown). Customerdatabase manager 708 is responsible for controlling and backing up thecustomer database and it also processes queries from customer accountmanager 710, service request manager 706, and results manager 712.Results manager 712 integrates the results generated by service requestmanager 706 with context-specific medical knowledge. Results manager 712receives information from search and filter engine 720 and mappingengine 722 specific to application requirements. Some results may beprovided as visual representations (e.g., mapped segmented structures ina brain image) while others may be expressed in numeric form (e.g.,coordinates of a neurosurgical target).

[0068] Preprocessor 714 checks patient data associated with a servicerequest to make sure that the patient data is in the correct format andthat the service request is appropriate. Support personnel 724 confirmthe check performed by preprocessor 714. Any inconsistencies oranomalies that are found are reported to service request manager 706.Similarly, post processor 718 checks the results of a service request.Support personnel 724 confirm the check performed by preprocessor 718.Any inconsistencies or anomalies that are found are reported to servicerequest manager 706.

[0069] Facility 700 also includes deformation engine 716, search andfilter engine 720, and mapping engine 722. Deformation engine 716computes atlas-to-patient transformations requested by service requestmanager 706. Search and filter engine 720 processes customer, patient,and procedure contextual data and integrates relevant atlas informationaccording to application requirements. Both results manager 712 andservice request manager 706 initiate search and filter engine 720operations. Following a request from results manager 712, mapping engine722 applies the results of the deformation process mapping an atlas tothe patient data.

[0070] Patient database manager 726 administers a database that containspatient data and corresponding transformations computed by deformationengine 716. Patient database manager 726 serves queries from deformationengine 716, search and filter engine 720, and mapping engine 722 and isalso responsible for controlling and backing up the patient database(not shown).

[0071] Atlas database manager 728 administers a database that containsthe atlas data (not shown). Atlas database manager 728 serves queriesfrom deformation engine 716, search and filter engine 720, and mappingengine 722 and is also responsible for controlling and backing up theatlas database. Atlas database manager 728 can also perform and/ormanage the indexing of the co-registered data databases.

[0072] While there has been illustrated and described what are atpresent considered to be preferred embodiments and methods of thepresent invention, persons skilled in the art will understand thatvarious changes and modifications may be made, and equivalents may besubstituted without departing from the scope of the invention.

[0073] In addition, many modifications may be made to adapt a particularelement, technique or implementation to the teachings of the presentinvention without departing from the central scope of the invention. Forexample, disclosed elements may be implemented in hardware, computerprogram code, or a combination of both hardware and computer programcode. Moreover, elements depicted and described separately may becombined and implemented in a single element. Therefore, this inventionis not limited to the particular embodiments and methods disclosed, butincludes all embodiments falling within the scope of the appendedclaims.

We claim:
 1. An apparatus having a mapping engine, a deformation engine,and a search and filter engine for producing composite data containingco-registered data and subject data, comprising: means for creating amapping relationship between the co-registered data and the subject databy mapping a template to the subject data; means for filtering theco-registered data; means for mapping the filtered co-registered data tothe subject data according to the mapping relationship to produce thecomposite data; and a means for transferring data, including the subjectdata and the composite data, wherein the transferring means includes adistributed communication controller interface.
 2. The apparatus ofclaim 1, wherein the distributed communication controller interfaceinitiates communications upon connection to a network utilizing acommunications protocol.
 3. The apparatus of claim 2, wherein thedistributed communication controller interface initiates thecommunications automatically.
 4. The apparatus of claim 2 wherein thecommunications protocol includes Jini.
 5. A method for presenting anoperator with composite data containing co-registered data and subjectdata, comprising the steps, performed in a client device of: requestingcomposite data from a composite data server having a distributedcommunication controller interface; transmitting the subject data tosaid composite data server through a distributed communicationcontroller interface in the client device; receiving the requestedcomposite data from the composite data server; and presenting thereceived composite data to an operator using a graphical user interfaceassociated with said a distributed communication controller interface inthe client device.
 6. The method of claim 5, wherein the presenting stepincludes the substep of: displaying said received composite data in abrowser having a graphical operator interface associated with saiddistributed communication controller interface in the client device. 7.The method of claim 5, wherein the requesting step includes the substepof: presenting the operator with a virtual control panel displayed in abrowser having a graphical operator interface associated with saiddistributed communication controller interface in the client device. 8.The method of claim 5, wherein the step of requesting comprises thesubstep of: issuing a request using a surgical navigation station with adistributed communication controller interface.
 9. The method of claim5, wherein the step of presenting comprises the substep of: respondingto a request using a surgical navigation station with a distributedcommunication controller interface.
 10. An apparatus for presenting anoperator with composite data containing co-registered data and subjectdata, comprising: means for requesting composite data from a compositedata server having a distributed communication controller interface;means for transmitting the subject data to said composite data serverthrough a distributed communication controller interface in the clientdevice; means for receiving the requested composite data from thecomposite data server; and means for presenting the received compositedata to an operator using a graphical user interface associated withsaid distributed communication controller interface in the clientdevice.
 11. The apparatus of claim 10, wherein the means for requestingcomprises: a surgical navigation station with a distributedcommunication controller interface.
 12. The apparatus of claim 10,wherein the means for presenting comprises: a surgical navigationstation with a distributed communication controller interface.
 13. Theapparatus of claim 10, wherein the distributed communication controllerinterface initiates communications upon connection to a networkutilizing a communications protocol.
 14. The apparatus of claim 13,wherein the distributed communication controller interface initiates thecommunications automatically.
 15. The apparatus of claim 13 wherein thecommunications protocol includes Jini.
 16. A method for producingcomposite data containing co-registered data and subject data,comprising the steps, performed in a server device having a mappingengine, a search and filter engine, and a deformation engine, of:creating a mapping relationship between the co-registered data and thesubject data by mapping a template to the subject data; filtering theco-registered data; mapping the filtered co-registered data to thesubject data according to the mapping relationship to produce thecomposite data; and transmitting the composite data through adistributed communication controller interface in the server device. 17.The method of claim 16, wherein the distributed communication controllerinterface initiates communications upon connection to a networkutilizing a communications protocol.
 18. The method of claim 17, whereinthe distributed communication controller interface initiates thecommunications automatically.
 19. The method of claim 17 wherein thecommunications protocol includes Jini.
 20. An apparatus for providingcomposite data to a client, comprising: a distributed communicationcontroller interface which receives client requests and subject dataover a network, wherein the distributed communication controllerinterface initiates communications upon connection to the network; asearch and filter engine operably coupled to the distributedcommunication controller interface, wherein the search and filter engineobtains template data from at least one co-registered database; adeformation engine operably coupled to the distributed communicationcontroller interface and the search and filter engine, wherein thedeformation engine produces a map relating coordinates of a subject dataspace to a co-registered data space; and a mapping engine operablycoupled to the deformation engine and the search and filter engine,wherein the mapping engine produces composite data which is transferredto a client using the distributed communication controller interface.21. An apparatus for presenting composite data to an operator,comprising: a user interface module for generating requests forcomposite data from a composite data server; a distributed communicationcontroller interface for transmitting subject data and the requests tothe composite data server over a network, and receiving composite datafrom the composite data server over the network; and a graphical userinterface module for presenting the composite data to an operator. 22.The apparatus of claim 21, wherein the distributed communicationcontroller interface initiates communications upon connection to thenetwork.
 23. The apparatus of claim 21, wherein the user interfacemodule further comprises: a probe tracked by a surgical navigationsystem, wherein the surgical navigation system includes a distributedcommunication controller interface.
 24. The apparatus of claim 21,wherein the graphical user interface module is associated with asurgical navigation system, wherein the surgical navigation systemincludes a distributed communication controller interface.